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Abstract. Context: concept Maps (CMs) enable the creation of a schematic representation of a 
domain knowledge. For this reason, CMs have been applied in different research areas, including 
Computer Science. Objective: the objective of this paper is to present the results of a systematic 
mapping study conducted to collect and evaluate existing research on CMs initiatives in Computer 
Science. Method: the mapping study was performed by searching five electronic databases. We 
also performed backward snowballing and manual search to find publications of researchers and 
research groups that accomplished these studies. Results: from the mapping study, we identified 
108 studies addressing CMs initiatives in different subareas of Computer Science that were re¬ 
viewed to extract relevant information to answer a set of research questions. The mapping shows 
an increasing interest in the topic in recent years and it has been extensively investigated due to 
support in teaching and learning. Conclusions: based on our results we conclude that the use of 
CMs as an educational tool has been widely accepted in Computer Science. 

Keywords: systematic mapping study; concept maps; knowledge representation; computer sci¬ 
ence; education tool. 


1. Introduction 


Managing knowledge is a key challenge for organizations and research institutions, 
since once knowledge is well managed, it can become a consistent intellectual com¬ 
ponent in the decision making process (O’Leary, 1998; O’Leary and Studer, 2001; 
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Zack and Serino, 2000). New challenges of how knowledge should be produced and 
transmitted have also been investigated (Correia, 2012). Among the strategies to work 
with knowledge, Concept Mappings (CMs) are considered an useful graphical tech¬ 
nique for organizing, understanding and sharing knowledge (Correia, 2012; Novak 
and Canas, 2008). 

Originally developed by Novak and Canas (2006) in 1972, CMs are a graphical 
representation of knowledge. CMs are visual representations of a particular topic and 
they provide support to organize and represent knowledge. CMs represent knowledge 
as graphs. These knowledge graphs are networks of concepts, where nodes represent 
concepts and links relations between them. These links contain associated words that 
describe the nature of the relationship between concepts (Novak and Canas, 2008). 

In summary, CMs provide a visual and holistic way to describe declarative knowl¬ 
edge relationships. They also provide an useful medium for sharing information in a 
concise and accessible form, making knowledge models based on CMs a promising 
vehicle for knowledge sharing (Novak and Canas, 2008). Originally CMs are connected 
to the education domain (Novak et al, 1984). CMs originated from the need to find how 
best to represent the conceptual understanding of children about science (Novak and 
Canas, 2006). It is possible to understand, from a simple search for related literature, 
that CMs are often applied as teaching and learning strategies (Gurupur et al., 2015; Jain 
et al, 2014; Katagall et al, 2015; Larraza-Mendiluze and Garay-Vitoria, 2013; Weinerth 
etal., 2014; Wu etal., 2012; Zvacek etal., 2012). CMs have been applied in pedagogical 
context in all disciplines including politics, philosophy of science, physics, chemistry, 
medicine, electrical engineering and mathematics (Katagall et al., 2015; Rueda et al., 
2009a). CMs have been used in multiple settings in education, since they are able to 
clarify the relationships between the subcomponents of a particular topic and they are 
useful to organize one’s own ideas (externalization) (Correia, 2012). 

Given the potential benefits of CMs, this paper aims to identify CMs initiatives in 
Computer Science by means of a systematic mapping study. A systematic mapping 
study, or simply mapping study, provides a broad overview of an area of research, to 
determine whether there is research evidence on a particular topic. Results of a mapping 
study helps identifying gaps in order to suggest areas for future research and provides a 
map that appropriately allows the position of new research activities (Kitchenham and 
Charters, 2007; Petersen et al., 2015). 

A mapping study presented in this paper investigates the following issues: 

(i) Purposes of applying CMs in Computer Science. 

(ii) Different subareas of Computer Science in which CMs have been applied. 

(iii) Resources used to support the development of CMs. 

(iv) Main guidelines used to evaluate CMs. 

(v) Benefits and problems reported on use of CMs in Computer Science. 

The rest of this paper is organized as follows. Section 2 presents related research. 
Section 3 discusses the research method applied to perform the mapping study. Results 
are presented in Section 4. Section 5 discusses the results, their implications, and limi¬ 
tations. Finally, Section 6 concludes the paper and presents directions for future work. 
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2. Related Work 

In general, secondary studies are based on analyzing research papers (referred to as 
primary studies). Mapping studies provide an overview of a topic area and identify 
whether there are subtopics where more primary studies are needed (Petersen et al., 
2015). In this paper we present a mapping study to identify and classify all research 
related to the use of CM in Computer Science. However, before accomplishing the 
secondary study presented in this paper, we performed a tertiary study looking for 
secondary studies that already investigated CMs in the same topic. Tertiary studies are 
considered as a review that focuses only on secondary studies, i.e., it is a review about 
other secondary studies (Kitchenham and Charters, 2007). In this tertiary study, we 
used the search string shown in Table 1. 

The search string presented in Table 1 was applied in the following electronic data¬ 
bases: IEEE Xplore, ACM Digital Library, Scopus, Science Direct and Compendex. We 
found only one study that drew our attention (Weinerth et al, 2014). 

In Weinerth et al. (2014), a literature review with a focus on usability was con¬ 
ducted. Authors conducted a research review to examine whether and how researchers 
have studied and considered usability when conducting computer-based CM assess¬ 
ments. From 119 journal studies that assessed computer-based CM only 24 discussed 
the usability issue in some way. According to Weinerth et al. (2014), the impact of 
usability on computer-based CM assessments is an issue that has received insufficient 
attention. 

In addition, we decided to look for secondary studies investigating CMs sepa¬ 
rately (not in the context of Computer Science). We used the search string shown in 
Table 2: 

The same seven electronic databases previously mentioned were searched, returning 
75 results. After eliminating duplications and applying the selection criteria, 19 second¬ 
ary studies were returned, but, none was a secondary study on CMs. However, one study 


Table 1 

Keywords of the Search String of the Tertiary Study on CMs in Computer Science 


Areas Keywords 


Concept Map “Concept Map”, “Concept Maps”, “Conceptual Map”, “Conceptual Maps”, “Concept 
Mapping” 

Computer Science “Computer Science”, “Computation”, “Software Engineering”, “Computer Engineering”, 
“Computing” 

Review “Systematic Literature Review”, “Systematic Review”, “Systematic Mapping”, “Mapping 

Study”, “Mapping Studies”, “Systematic Literature Mapping”, “Literature Review” 

Search String: (“Concept Map” OR “Concept Maps” OR “Conceptual Map” OR “Conceptual Maps” 

OR “Concept Mapping”) AND (“Computer Science” OR “Computation” OR “Computer 
Engineering” OR “Software Engineering” OR “Computing”) AND (“Systematic Literature 
Review” OR “Systematic Review” OR “Systematic Mapping” OR “Mapping Study” OR 
“Mapping Studies” OR “Systematic Literature Mapping” OR “Literature Review” ) 
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Table 2 

Keywords of the Search String of the Tertiary Study on CMs 


Areas Keywords 


Concept Map “Concept Map”, “Concept Maps”, “Conceptual Map”, “Conceptual Maps”, “Concept 
Mapping” 

Review “Systematic Literature Review”, “Systematic Review”, “Systematic Mapping”, “Mapping 

Study”, “Mapping Studies”, “Systematic Literature Mapping”, “Literature Review” 

Search String: (“Concept Map” OR “Concept Maps” OR “Conceptual Map” OR “Conceptual Maps” OR 

“Concept Mapping”) AND (“Systematic Literature Review” OR “Systematic Review” 
OR “Systematic Mapping” OR “Mapping Study” OR “Mapping Studies” OR “Systematic 
Literature Mapping” OR “Literature Review”) 


attracted our attention. Rosas and Kane (2012) did not present a secondary study, but 
they conducted a pooled analysis of 69 CM studies to describe characteristics across 
study phases, generated specific indicators of validity and reliability, and examined the 
relationship between selected study characteristics. A majority of CM studies analyzed 
were classified as directed to public health (59.4%). Others were in the fields of human 
services (20.3%), biomedical research (5.8%), social science research (2.9%), and busi¬ 
ness or human resources (2.9%). With respect to purpose, twenty-eight (40.6%) were 
used for strategic planning purposes. 

Based on the results from this investigation by employing tertiary studies, we can say 
that there is only one recent extensive literature search that focused on computer-based 
assessment, CMs and usability. This motivated us to conduct this study. 


3. Mapping Study 

As previously mentioned mapping studies build a classification scheme and organize a 
research field of interest within the categories of the scheme. As a result, the coverage of 
the research field can be determined and the scheme can also be used to answer specific 
research questions (Petersen et al., 2015). Therefore, mapping study provides an over¬ 
view of a research area by identifying and quantifying the related available research. 

The research method for the mapping study presented in this paper was defined based 
on the guidelines given by Kitchenham and Charters (2007) and Petersen et al. (2015). 
This mapping involves three main phases: 

(i) Planning: refers to the pre-review activities, and aims at establishing a review 
protocol defining the research questions, inclusion and exclusion criteria, sources 
of studies, search string, and mapping procedures. 

(ii) Conducting: searches and selects the studies, in order to extract and synthesize 
data from them. The data extraction activity for a mapping study is broad and the 
analysis of a mapping does not include the use of in-depth analysis techniques, 
such as meta-analysis, but rather total amounts and summaries. Graphic represen¬ 
tations can also be used to summarize the data. 
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(iii) Reporting: final phase that aims at writing up the results and circulating them to 
potentially interested parties. In this phase the findings of the mapping study are 
used to answer the research questions. 

Kitchenham and Charters (2007) suggest, in addition to the searches in the databases, 
conducting a snowballing from reference lists of selected studies, in order to identify addi¬ 
tional relevant studies through the reference lists of the studies found using the search strings 
(Jalali and Wohlin, 2012). Manual searches for works developed by important researchers 
and research groups can also be performed (Kitchenham et al., 2011). These researchers/ 
research groups are identified from previously selected studies (retrieved by searches in the 
databases, as well as from snowballing). We tried using these two approaches in our map¬ 
ping to overcome the limitation of using a specific set of electronic databases. 

The following are the main steps we performed for the mapping study. 


3.1. Research Questions 

This mapping study was conducted in order to identify principles of CMs initiatives in 
Computer Science. Based on this goal, seven research questions (RQs) were created. 
Table 3 presents the RQs that this mapping study aims to answer, as well as the rationale 
for considering them. 


Table 3 

Research questions and their rationales 


No Research question 


Rationale 


RQ1 When and where have the studies been The topic of this mapping study seems to be broad and 
published? new. This research question provides an understanding 

on whether there are specific publication sources for these 
studies, and when they have been published. 

RQ2 What is the purpose of the application of This question looks for the purposes declared in the studies 
CMs in Computer Science? for applying CMs. This is important to point out why such 

studies have been accomplished. 

RQ3 In which different subareas the application Investigates the different subareas of Computer Science in 
of CMs have been presented? which CMs have been applied. 


RQ4 


What are the main instruments used to 
develop the identified CMs? 


Highlights the main instruments, as tools, techniques or 
methods, currently used to develop CMs. This is useful for 
researchers and practitioners that intend to accomplish new 
studies in this context, as well as to guide future research 
towards new technologies in order to fill the existing gaps. 

RQ5 What are the guidelines or mechanisms This RQ highlights guidelines or mechanisms used to 


used to evaluate the CMs identified? 


evaluate the CMs. Similarly to the previous one, this is 
useful for researchers and practitioners to guide future 
research. 


RQ6 What are the benefits reported by authors Provides an overview of the main benefits reported by 
related to the use of CMs? authors related to the main advantages in works with CMs. 

RQ7 What are the problems reported by authors Provides an overview of the main limitations reported by 
related to the use of CMs? authors related to the main difficulties in works with CMs. 
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3.2. Search Strategy and Selection 

The purpose of a mapping study is to conduct a review of relevant studies to assess the 
quantity of evidence existing to address the RQs. The process needs to be rigorous and 
unbiased and it often involves a wide coverage of sources, such as online electronic 
databases, journals and conferences. In order to minimize bias and to maximize the 
number of sources examined, a pre-defined strategy to identify potential primary studies 
is required. Ours is described as follows. 

3.2.1. Terms and Search String 


First of all, we created our search string. The search string considers two areas - “Con¬ 
cept Map” and “Computer Science” (Table 4) - and it was applied in three metadata 
fields: title, abstract and keywords. The search string went through syntactic adaptations 
according to particularities of each source. 

In order to elaborate the search string, the set of search terms were devised in an 
iterative way, i.e., we started with an initial set of terms and iteratively improved this set 
until all relevant pre-selected studies were found. The pre-selected studies are part of a 
control group. If the publications of the control group are not retrieved, the string needs 
to be calibrated (adjusted) or new databases need to be added in order to guarantee that at 
least these publications are retrieved. In this sense, we used a control group to calibrate 
the search string. 

3.2.2. Sources 

We chose the electronic databases based on predefined criteria, such as: (i) coverage 
(large number of conferences proceedings, journals, and indexed books in different 
knowledge areas); (ii) content update (publications are regularly updated); (iii) avail¬ 
ability (full text of the study are available); (iv) quality of results (accuracy of the results 
obtained by the search); and (v) versatility export (a mechanism to export the results of 
the search is available). The control group, described before, also helped to define the 


Table 4 

Keywords of the Search String of the Mapping 


Areas Keywords 


Concept Map “Concept Map”, “Concept Maps”, “Conceptual Map”, “Conceptual Maps”, “Concept 

Mapping” 

Computer Science “Computer Science”, “Computation”, “Software Engineering”, “Computer Engineering”, 
“Computing” 

Search String: (“Concept Map” OR “Concept Maps” OR “Conceptual Map” OR “Conceptual Maps” 

OR “Concept Mapping”) AND (“Computer Science” OR “Computation” OR “Software 
Engineering” OR “Computer Engineering” OR “Computing”) 
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databases. The most commonly used databases in Computer Science area that meet to 
the characteristics described above are: 

IEEE Xplore (http://ieeexplore.ieee.org) 

ACM Digital Library (http: //dl. acm. org) 

ScienceDirect (http: //www. sciencedirect. com/) 

Scopus (http://www.scopus.com/) 

Compedex (http://www.engineeringvillage2.org) 

As a result of searching from the search string defined and the selected sources, a 
total of 700 publications were returned. We considered the studies published until June 
2016. 

3.3. Selection Process 

The selection criteria are organized in one Inclusion Criterion (IC) and seven Exclusion 
Criteria (EC). The following inclusion criteria were applied: 

(IC1) The study must be within the Computer Science and discuss the CMs initia¬ 
tive. 

(EC1) The study does not have an abstract. 

(EC2) The study is just published as an abstract. 

(EC3) The study is not written in English. 

(EC4) The study is an older version of other study already considered. 

(EC5) The study is not a primary study, such as editorials, summaries of keynotes, 
workshops, and tutorials. 

(EC6) The study is not in the scope of Computer Science, although it uses CMs 
initiatives. 

(EC7) The study is in Computer Science context, however, does not use CMs initia¬ 
tives. 

The selection process was divided into five stages, as shown in Fig. 1. 

In the 1 st stage duplicated studies were eliminated, resulting in 367 publications (re¬ 
ducing approximately 47.6%). 

In the 2 nd stage, the selection criteria (inclusion and exclusion criteria) were ap¬ 
plied for title, abstract and keywords, leading to 158 studies (reducing approximately 
56.9%). 

In the 3 rd stage, the selection criteria were applied considering the full text, resulting 
in a set of 93 studies (reducing approximately 41.1%). Over these 93 studies consid¬ 
ered relevant, we performed backward snowballing, which resulted in 251 studies. From 
these 251 studies, the selection criteria were applied considering the title, abstract and 
keywords. Next, the selection criteria were applied considering the full text. A total of 
37 studies were resulted from this stage. 

Finally, from 130 studies selected until then (93 from sources and 37 from snow¬ 
balling), we looked for publications authored by the researchers and research groups 
involved in these studies. We searched for their personal pages as well as other publica¬ 
tions authored by them in electronic databases that we used as sources for this mapping. 
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Fig. 1. Search and selection mapping process. 


A total of 52 studies were selected. After the selection criteria were applied, only 1 study 
remained. As a result, we got to 131 studies (93 from the sources, 37 from snowballing, 
and 1 from direct search to researchers and research groups). 

In the 4 th stage, we applied Visual Text Mining (VTM) techniques in order to ensure 
that relevant studies were not eliminated (Felizardo et al, 2012). One VTM technique is 
to create a content map, i.e., a two-dimensional (2D) visual representation, where each 
document (i.e., a primary study) is graphically represented as an element on the plane, 
shown graphically as a circle. Positions of the studies in the map reflect the similarity 
relationships between their content (title and abstract). Therefore, if the studies are more 
similar (exhibiting content similarity) they are shown closer to one another; conversely, 
more dissimilar, further apart they are in the graph. 

In summary, the strategy suggested by Felizardo et al. (2012) to review the studies’ 
selection activity is to analyze the content map in terms of the included and excluded 
studies to find inconsistencies, i.e., studies with similar content (meaning that they are 
positioned close together on the map) with different classifications: included and exclud¬ 
ed. These cases are hints to the reviewer, and the studies should be reviewed following 
the traditional method, i.e. reading the full text. Fig. 2 shows the clustering generated 
from the 131 studies. For example, the study highlighted with an arrow (see Cluster 7 in 
Fig. 2) was reviewed and its classification was changed from excluded to included. This 
same procedure was adopted for all clusters. In total, 6 studies initially excluded were 
included after the review, totaling 137 studies. 
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Fig. 2. Content map generated from 131 studies. 


In order to analyze the selected 137 studies, we conducted a quality assessment. This 
step allowed us to perform a 5 th stage in the selection process based on detailed quality 
criteria. In order to analyze the quality of the included primary studies we developed a 
checklist, as a quality instrument, containing seven questions (Table 5). The objectives 
of this list was to: identify the quality of the study regarding the main topic of this map¬ 
ping and to identify the quality of the information that the study provides us, in order 
to know the relevance of the study within the SM. Our checklist was adapted from the 
generic quality criteria created by Kitchenham Kitchenham (2004). For each question in 
the checklist, the following scale-point was applied: Yes - 1 point; No - 0 points. The 
total quality score therefore fell into the range between: 0-1 (very poor); 2 (fair); 3-4 
(good); 4-6 (very good) and 7 (excellent). 


Table 5 

Study Quality Checklist 


No Item Answer 

Q1 The CM was evaluated? Yes/No 

Q2 Is it clear how the approach to evaluate CM was applied? Yes/No 

Q3 Is it clear in which Computer Science topic, CM was applied? Yes/No 

Q4 Did the study show improvements in CM construction or a new way to construct it? Yes/No 

Q5 Was CM generated with a tool support? Yes/No 

Q6 Are the benefits clear in using CM? Yes/No 

Q7 Were there limitations in using CM? Yes/No 
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The quality score was used to decide the inclusion or exclusion of a primary study. 
The cutoff scale-point, selected by us, to exclude studies from the list of selected studies 
was 2.0 (very poor or fair quality). A total of 29 studies were excluded from this stage. 
As a final result, we got to 108 studies to be analyzed. Appendix A presents the reference 
of the selected studies. 


3.4. Data Extraction and Synthesis 

We considered different facets for classifying the studies. The facets were defined ac¬ 
cording to the RQs and taking the selected studies into account. So, depending on the 
focus of each scheme, the study was classified as one or any combination of this. Fol¬ 
lowing, the categories of these facets are presented. 

Purposes (RQ2): in this facet, we wanted to learn the purposes, when applying CMs in 
Computer Science. We have identified five main categories of such purposes: 

• Learning Level Evaluation: it is related to the academic environment and quan¬ 
tifies how much knowledge was absorbed by a person from the use of CMs. 

• Assistance in Teaching and Learning: this classification is related to the initia¬ 
tives that use CMs as an educational tool in Computer Science. 

• Knowledge Organization: from the structure of a CM, stakeholders can easily 
navigate through desired topics. This category is related to the initiatives that use 
CMs as a means to organize any knowledge on a particular subject. 

• Software Visual Representation: this classification is designed to incorporate 
the initiatives that use CMs as a conceptual model for software requirements that 
assist stakeholders in the development, evaluation and maintenance of software. 

• General: this category is used to classify those studies that discuss CMs in Com¬ 
puter Science in general. 

Subareas applied (RQ3): this facet refers to the different subareas that CMs have been 
applied in Computer Science. We have identified seven main subareas: (i) Artificial Intel¬ 
ligence; (ii) Software Engineering; (iii) Database; (iv) Ontologies; (v) Human-Computer 
Interaction; (vi) Programming Languages; and (vii) Not Specified or General. 

Resources (RQ4): this facet discusses the different resources used to support the de¬ 
velopment of CMs in Computer Science, as tools, techniques or methods. Based on the 
selected studies, we considered the following categories: (i) XML Schema; (ii) Text 
Mining; (iii) Paper and Pen; (iv) Ontologies; (v) Own System; (vii) Intelligent Knowl¬ 
edge System; (vii) Data Ink; (viii) Collaborative Construction; (ix) Concept Map Editor; 
and (x) Not Mentioned. 

Evaluation (RQ5): this facet discusses the main guidelines or mechanism used to evalu¬ 
ate the CMs identified in the included studies. We adopted the following classification: 
(i) Support of a software for comparison; (ii) Structural with expert map; and (iii) Rela¬ 
tional with expert map. 
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Reported benefits (RQ6): the categories for this facet are based on the main benefits 
related to use of CMs reported in the included studies. We have identified four main cat¬ 
egories of benefits, namely: (i) Flexible structure and easy to understand; (ii) Knowledge 
representation; (iii) Knowledge Sharing; and (iv) Pedagogical contribution. 

Reported Problems (RQ7): the categories for this facet are based on the main prob¬ 
lems highlighted by the authors of the included studies related to use of CMs in Com¬ 
puting Science. We have identified four main categories of problems, namely: (i) Dif¬ 
ficulty in creating the CMs; (ii) Cognitive overload; (iii) Increased workload; and (iv) 
Scalability. 


4. Results of the Mapping 

The results of our mapping study are presented. We used the facets of the classification 
schema aforementioned to answer the RQs. Appendix B presents the mapping of indi¬ 
vidual studies and its categories. 


4.1. Frequency of Publication (RQ1) 


In order to offer a general view of the efforts in the area of CMs in Computer Science, a 
distribution of the 108 selected studies over the years is shown in Fig. 3. As this figure 
suggests, although the research of CMs in Computer Science is not recent, the interest 
in CMs has been moderately increasing within 2006-2015, with a significant increase 
in 2011 and 2012. 

Looking at the publication vehicle, conferences seem to be the main communication 
channel representing 54,62% (59 studies) of publications. Journals occupied 35.1% (38 
studies), workshops 8,3% (9 studies) and finally symposium with 1.8% (2 studies). 


14 



Fig. 3. Distribution of the selected studies over the years. 
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4.2. Purposes of CMs (RQ2) 

Fig. 4 shows the number of studies per purposes of application. We can notice that 
Assistance in Teaching and Learning (63 studies - 58.33%) and Knowledge Organiza¬ 
tion (40 studies - 37%) have the largest representativeness. Some purposes that were 
identified are strongly related. For instance, knowledge organization aims to support as¬ 
sistance in teaching or learning. Thus, many studies were considered in both categories. 
Furthermore, many studies assigned in those categories use CMs as a means of teaching 
undergraduate courses of Computer Science and Software Engineering. CMs are ef¬ 
fective educational tools which consist of concepts (Atapattu et al., 2015a) and expert¬ 
generated CMs can be used to guide students as in a content repository (Williams et al., 
2012). Maps can improve learning when used as an advanced organizer in the classroom 
(Faily et al., 2012). 

CMs can also be employed more effectively to determine the depth of knowledge 
gained by a student. A total of 19 studies (approximately 17,6%) have as a purpose 
Learning Level Evaluation. According to Gary et al. (2012), CMs offer an alterna¬ 
tive assessment tool that can be used to show how much a student may have acquired 
knowledge. 

Even with a smaller percentage of studies, CMs have also been used for Software 
Visual Representation (7 studies - 6.5%) and it presented interesting results in soft¬ 
ware development. For instance, CMs have been used in object-oriented modeling 
(Sien, 2010), Software Requirements (Coffey et al., 2012; Faily et al., 2012; Kof 
et al., 2007), Software Security Assurance (Snider et al., 2014), process of developing 
software systems (Gurupur et al., 2014) and Software Assessment (Williams et al., 
2012 ). 


Learning Level Evaluation 
Assistance inTeachingand Learning 
Software Visual Representation 
Knowledge Organization 
General 



0 10 20 30 40 50 60 70 

NUMBER OF STUDIES 


Fig. 4. Purpose of applying CMs in Computer Science. 
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4.3. Subareas Applied (RQ3) 

Fig. 5 shows the number of studies considering the different subareas in computing sci¬ 
ence. Note that most of the studies (76 studies - 70.4%) did not clarify to which com¬ 
puting science sub-area the work is related. The studies discuss initiatives with CMs in 
computing science in a general form. For example, the studies (Alvarez Bermejo et al., 
2015; Atapattu et al., 2015a; Huang et al., 2012) argue that the use of CM helps to con¬ 
ceptualize, schematize and present procedural information like software documentation, 
explaining laboratory reports, in classroom. The studies argue how the techniques for 
representing knowledge, as CM, can be effectively used in a classroom for the under¬ 
standing of disciplines in computing science. 

Looking at the studies that present initiatives in a specific sub-area of Computer Sci¬ 
ence, two categories have greater representation: 

• Programming Languages (10 studies - 9.6%). 

• Software Engineering (12 studies - 11.1%). 

In the Programming Languages category, CMs have been used mainly to present the 
Object-Oriented Programming (OOP) paradigm. For example, (Berges and Hubwieser, 
2013; Berges et al., 2012; Dogan and Dikbiyik, 2016; Hubwieser and Miihling, 2011), 
presented results investigating how CMs have been used in OOP teaching and learning. 
Table 6 presents the other programming languages. 

In relation to Software Engineering, CMs have been applied with the purpose of as¬ 
sisting in different activities that involve software development process (Coffey et al., 
2012; Faily et al., 2012; Upchurch and Sobel, 2002; Williams et al, 2012), as well as 
how to organize concepts in Software Engineering domain (Gurupur et al., 2014; Koch 
and Landes, 2015; Leake and Wilson, 2001; Mohamed et al., 2013). In order to demon¬ 
strate this purpose, some studies are described below. 
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Fig. 5. Subareas that CMs in Computer Science have been applied. 
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Table 6 


Studies per Programming Languages 

Programming 

Languages Studies 

Object-Oriented Programming 
(OOP) 

C Programming 

Algorithm and Data Structures 

Sien (2010); Berges etal. (2012); Dogan and Dikbiyik (2016); Hubwieser and 
Muhling (2011); Berges and Hubwieser (2013); Jain et al. (2014) 

Matthews (2010); Harrow et al. (2007) 

Burdescu et al. (2008); Florea and Radu (2007) 


In Coffey et al. (2012); Faily et al. (2012); Kof et al. (2007), for instance, CMs were 
used to make sense of and improve the quality of a software requirements specifica¬ 
tion and representation of traceability between requirements. In Gurupur et al. (2014) 
a process of developing software systems is described by capturing the conceptual do¬ 
main knowledge of the problem domain using CMs. According to Gurupur et al. (2014), 
there is a gap between user’s needs and software implementation efforts. This problem 
persists mainly because software systems often do not fully support user needs. Leake 
and Wilson (2001) describe a research on supporting aerospace design by integrating a 
case-based design support framework with interactive tools for capturing expert design 
knowledge through CMs. 

Regardless of the sub-areas in which CMs are used, it is possible to note that the 
main motivation for integrating CM is the fact that it makes easy to represent and under¬ 
stand a specific domain. 


4.4. Resources (RQ4) 

In this RQ we investigated what were the resources used in the studies to support the 
CMs construction. Nine different resources are highlighted in this context. Fig. 6 shows 
the number of studies considering the different resources used to construct CMs. 

The great majority of the studies use their own systems (58 studies - 53.7%). 
The studies in this category propose the construction of a new system to assist in the 
creation of CMs in order to share and visualize knowledge. Many of the studies in 
this category suggest the construction of a tool for building CMs collaboratively (15 
studies - 13.9%). Recent approaches of teaching and learning put emphasis in activi¬ 
ties that take place in a collaborative frame. Collaborative learning approaches seem 
to encourage knowledge construction and it can lead to better development of ideas 
and concepts through discussion and negotiation. For instance, Tomoto et al. (2011) 
presented a collaboration tool for knowledge integration and sharing in cross-cultural 
communication using a participant-constructed collaborative CM. MaGIA tool was 
presented in Blecic et al. (2007), an Internet-based multi-user system designed for a 
collective and collaborative construction of knowledge models represented as CMs. 
Martinez-Maldonado et al. (2015) described the Tabletop-Supported Collaborative 
Learning (TSCL), a conceptual model that provides foundations for building tabletop- 
based systems that can inform understanding of the collaborative learning process. 
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Fig. 6. Resources used to support the CMs construction. 


Other collaborative tools are: Canas and Hill (2010); Chung et al. (2010); Harrow 
et al. (2007); Herl et al. (1999); Hui (2003); Komis et al. (2002); Kremer and Gaines 
(1994); Kristensen (2011); Lahti (2011); Marshall et al. (2003); Tegos et al. (2015); 
Zanetti et al. (2006). 

Another resource used by many studies for the construction of CMs is the Concept 
Map Editor (Cmap Tools 1 ) (34 studies - 31.4%). Cmap is a traditional tool, commonly 
used to support the construction, navigation and sharing knowledge models represented 
through CMs. Besides providing basic operations to draw and label CMs, CmapTools 
includes extensive capabilities, for example, to link CMs with electronic resources, such 
as, images, diagrams, video clips, and other CMs, enabling the construction of richly 
connected concept-map-based knowledge models for particular domains. It also enables 
distributed storage and access to CMs on multiple servers, to support knowledge sharing 
across multiple sites (Canas et al., 2004). 

Besides the two most used resources, presented above, it is interesting to note the di¬ 
versity of other resources used to support the construction or manipulation of CMs. For 
example, the creation of CM based on an existing ontology; the extraction of concepts 
and relationships from a text to automatically generate the CMs (called concept mining 
map); parser in XML-based document to extract concepts and their associated relations; 
and the traditional way of constructing CMs using paper and pencil. The studies in each 
one of these categories are presented in Appendix B, Table B3. 


4.5. Evaluation of CMs (RQ5) 


One of the main issues related to CMs is how to evaluate them. Several scoring schemes 
for human-based evaluation have been proposed. The analysis presented in Anohina and 


http://cmap.ihmc.us/ 
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Grundspenkis (2009) shows that most of them are based on quantitative measures, that 
is, structural analysis (number of valid propositions, levels of hierarchy, etc.)- Another 
method is to compare CMs created with one or more experts’ maps. Currently, there are 
also proposals for automated evaluations of CM. 

In this facet we analyze whether the CMs presented in the 108 studies that were left 
in this mapping were evaluated. Only 38 (35.1%) presented some type of evaluation. 
Then, we classified the main mechanism used to evaluate CMs identified in the included 
studies. Three types of evaluation were identified: (i) Relation with expert map; (ii) 
Structural analysis; and (iii) Support of a software to compare. Fig. 7 shows the number 
of studies considering the different types of evaluation used. 

The first classification relates to evaluate the CM comparing with an expert’s CMs. 
Normally, a CM which was developed by an expert is used as reference for evaluating 
other CMs created later. The great majority of the studies included in this mapping 
used this classification to evaluate the CMs generated (23 in 38 studies, corresponding 
to 84.2%). In Grundspenkis (2008), for instance, an approach is presented in which 
ontologies are transformed into CMs to be used for systematic creation of learner 
knowledge structure. In order to evaluate the CM created, a knowledge evaluation 
agent compares the learner’s CM with a teacher’s map and corresponding ontology, 
and recognizes synonyms and several patterns (correct or incorrect) of learner’s solu¬ 
tion. Rigby et al. (2009) examined means of increasing learners’ expertise of Com¬ 
puter Information Technology course about the concepts through the use of varying 
types of representations, as CMs. The students created CMs of their understanding 
of threat analysis, which was then scored against an expert CM. Finally, Qasim et al. 
(2013) proposed an algorithm for automatic candidate term extraction from text docu¬ 
ments to construct a CMs. CMs were verified by experts with doctoral degrees in the 
field of Information System. 

The second classification refers to the structural analysis consisting of the compari¬ 
son of CMs considering the number of valid propositions, levels of hierarchy, number 
of concepts, etc. In Gurupur et al. (2015) a tool is described that can be effectively 
used to evaluate student learning outcomes using CMs and Markov chain analysis. The 
CMs created are compared with each other from the calculation of number of concepts 
in hierarchy, number of support level concepts and number of detail level concepts. In 
Arruarte et al. (2014) a study was conducted using the concept mapping technique in 
computer engineering education for learning theoretical introductory database topics in 


Relation with expert map 
Structural analysis 
Support computational to comparison 
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Fig. 7. Evaluation of CMs. 
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a collaborative way. Some sources of data have been analyzed to evaluate the results of 
the study, as individually created CMs and collaboratively created. CMs were evaluated 
in terms of the average number of nodes, average number of nodes relationships, com¬ 
pleteness and thoroughness. 

The third classification refers to evaluate CMs with software support. In Larraza- 
Mendiluze and Garay-Vitoria (2013), for instance, a process is presented to analyze CMs 
created by students about computer structure subject. All the CMs were merged to show 
the relationships that were missing. Another example of this category can be observed in 
Vaishnavi et a\. (2016). The CMs elaborated by students are converted to a XML format 
and concepts and their relationships are identified. A probability distribution of concepts 
in the CM developed is calculated. The normal distribution is then represented as a form 
of learning curve. The evaluation of student’s level of understanding content is assessed 
by analyzing the learning curve generated by this approach. 


4.6. Reported Benefits (RQ6) and Problems Using CMs (RQ7) 


As we can see from this mapping study, there are many benefits of implementing CMs 
in Computing Science: 

• Flexible structure and easy to understand: CMs are a simple graphical repre¬ 
sentation and it is easy to understand a link between different concepts. 

• Knowledge representation: knowledge represented in a visual format makes 
easier the recognition and the understanding of incoming information. In many 
disciplines various forms of CMs are already used as formal knowledge repre¬ 
sentation systems, for example: semantic networks in artificial intelligence, bond 
graphs in mechanical and electrical engineering, Petri nets in communications, 
requirements in the software development process. 

• Knowledge sharing: researchers have investigated CMs for collaboration 
through map sharing. Semantic integration has received a significant amount 
of attention in recent years, particularly from database researchers seeking to 
facilitate information sharing and integrate heterogeneous data sources (Kris- 
tensen et al., 2009). 

• Pedagogical contribution: in educational area CMs can be used for supporting 
learning and also for assessing learning (Arruarte et al., 2008). It is possible 
to note from our results (Fig. 8) that there are many studies that dealt with the 
effect evaluation of CMs on students’ achievements in a wide diversity of ar¬ 
eas, specially in Computer Science. According to Gurupur et al. (2015), CMs 
provide visual data to instructors on student misconceptions and their level of 
understanding. Hence CMs can be helpful to develop certain abilities such as, 
drawing reasonable inferences by identifying key concepts on a topic, synthe¬ 
size and integrate the information and ideas, and learn concepts and theories in 
a study area. 

Fig. 8 shows the main benefits in using CMs. The pedagogical contribution through 
CMs appears most representative. The fact that CMs can be used for different pedagogi- 
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Fig. 8. Main benefits identified from CMs. 


cal purposes, together with the nature of Computer Science, its many facets representing 
different abstraction levels and the interconnections among them, makes CMs relevant 
for Computer Science teaching and learning processes in general. Interestingly, despite 
the use of CM bringing many benefits, some authors, even though few, also highlight 
problems that may arise from the use of CM, such as: 

• Difficulty in creating the CMs: it is important to have the knowledge domain on 
the subject to develop a CM with quality. 

• Cognitive overload: in terms of the short-term memory, CM has a weakness; it 
might contain too many concepts in its structure causing learner’s cognitive over¬ 
load, i.e., becomes difficult to understand the content wanted to be learned. 

• Increased workload: shortage of time is a potential risk to incorporate the use of 
CMs when working with deadlines, for example, in software development compa¬ 
nies. The development or learning of another artifact can imply in increasing the 
employee workload and costs. However, automatic or semi-automatic creation of 
CMs from text documents is a worthwhile research very investigated nowadays 
(Chen et al, 2008; Qasim et al, 2013). 

• Scalability: CMs with large scale quickly become too complex to be processed by 
the learner. A representation of a CM can lose its utility when the amount of infor¬ 
mation increases. The high number of concepts and cross-links between concepts 
makes difficult to understand the represented knowledge. 

Fig. 9 shows how the main problems aforementioned are perceived in the included 
studies. It is important to highlight that not all studies mention problems related to the 
use of CMs. Cognitive overload was the problem more cited in the studies. In fact, 
cognitive load can becomes a problem when the domain of application is large, and con¬ 
sequently, CMs become overburdened with too much information and complexity (Wil¬ 
liams et al., 2012). According to Williams et al. (2012), this phenomenon (termed “map 
shock”) results in student disengagement and reduction of expected learning gains. 



Analyzing the Use of Concept Maps in Computer Science :... 


275 



Fig. 9. Main problems identified from CMs. 


5. Discussion 

CMs were first proposed in education, to enable students to externalize their knowl¬ 
edge by constructing a based-representation of concepts and their relationships. This 
representation has been considered as a cognitive structures suitable for assessment or 
knowledge sharing. CMs are used worldwide to facilitate knowledge examination, con¬ 
struction, comparison, and reuse by users ranging from elementary graduated students 
to researchers. In the Computer Science area, this is no different. The main contribution 
of this research is to evidence how CMs have been used in Computer Science. This sys¬ 
tematic mapping has shown that CMs have been heavily used with different purposes 
and in different subareas. 

Within the Computer Science context, several studies reported that the CMs have 
been used, for example, to support artificial Intelligence, Human-Computer Interaction, 
Programming Languages and Software Engineering (Berges etal., 2012; Kot'e/ al., 2007; 
Leon et al., 2008; Portmann et ah, 2012). Many of the studies included in this mapping 
use CMs on general Computer Science subjects, that is, the subarea is not specified. On 
the other hand. Software Engineering and Programming Languages, although with few 
studies in the literature, were the subareas that stood out the most. It is possible to iden¬ 
tify that CMs can provide important benefits in the representation of some programming 
languages paradigms (OOP, C), in the development of software product, in improving 
the requirements specifications quality or in software requirements traceability (Coffey 
et al., 2012; Gurupur et al., 2014). 

Considering the two most outstanding subareas, Software Engineering and Program¬ 
ming Languages, we did an analysis of the year in which the studies were published. 
Software Engineering, for example, out of the 11 articles in this category, eight were 
published from 2009, and 3 in 2013. This shows that the interest in CM in software en¬ 
gineering can be considered recent. Similarly, Programming Language presents itself as 
a subarea that with interest in using CM recent, starting in 2003 to 2016. 
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The subareas were also analyzed according to their purposes (RQ2). The selected 
studies of Software Engineering have as main purposes the Knowledge Organization 
and Software Visual Representation with 81.8% of representatively. For example, in 
Gurupur et al. (2014), semantic requirements sharing approach is shown to reduce the 
semantic gap involved in the process of developing software systems. According to Gu¬ 
rupur et al. (2014), sometimes a requirements engineer may not fully comprehend the 
problem domain. Therefore, a CM allow to build a schematic representation of the do¬ 
main knowledge and its can be easily comprehended by every individual involved in the 
development of a software system. 

The studies selected in the Programming Language subarea, six of the ten studies 
has the purpose of Learning Level Evaluation. This result makes sense, since CMs have 
been proposed as a potentially facilitative learning strategy. According to Hubwieser and 
Muhling (2011), in science education, there are many research activities that use CM in 
order to investigate cognitive structures. Consequently, CM can be used to measure the 
knowledge level and/or learning in programming disciplines. 

We analyzed in the selected studies in order to identify which were the tools used to 
construct CM to represent programming language, for example. The same happens in 
other categories. The main tool used in the programming language subarea were proper¬ 
ty systems. CMs have been created mainly by systems or applications developed by the 
authors that wrote the papers. This happens because tools available in the literature do 
not always meet the different purposes of each study. This causes researchers to create 
a new tool or customize an existing tool. In addition, it is important to note the priority 
that studies have given to CM construction in a collaborative way from the tools created. 
According to Tomoto et al. (2011), our world becoming globalized needs no borders but 
must be opened to cross-cultural, collaborative communication in every social field, in¬ 
cluding education. International distance education could more broadly help participants 
experience concept construction in the distance communication environment. Techno¬ 
logical contributions can contribute to achieving these goals, adding a new dimension to 
collaborative learning. 

One result that attracted attention was in relation to the evaluation of CMs. Al¬ 
though there are in the literature numerous systems or methods of CM evaluation, only 
35.1% of the CMs included in this mapping were evaluated. The minority of included 
studies in this mapping (38 of 108) reported some CM evaluation, being: Relation with 
expert map, structural analysis and support of software to compare are examples of 
evaluation types. 

In addition to the benefits already mentioned about the use of CMs, we argue that 
they are also useful in secondary studies context, discussed as follows. Many research¬ 
ers conducting secondary studies, as one of their first activities, need to read the studies 
abstracts to decide about their inclusion or exclusion. In some domains, such as clinical 
Medicine, Education, Psychology and Social Sciences, the review of abstracts is more 
than enough to determine its relevance (Kitchenham, 2004). However, in Computer Sci¬ 
ence, especially, in Software Engineering, there are unstructured abstracts in the studies. 
This becomes difficult to identify the relevance or not of the studies (Brereton et al., 
2007). Recent literature has provided evidence that unstructured and poorly written ab- 
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stracts may compromise the selection activity (Brereton et al., 2007; Dyba and Ding- 
soyr, 2008; Dyba et al., 2007; Kitchenham, 2008; Petersen and Nauman, 2011; Zhang 
and Muhammad, 2012, 2011a,b). 

One potential solution to minimize such problem, experienced during a secondary 
study selection, is to promote the use of structured and graphical abstracts in Software 
Engineering (Hartley, 2004; Kitchenham, 2008). Since CMs can help in the understand¬ 
ing of a study, our research group have investigated the creation of graphical abstracts 
to support the selection activity in secondary studies using CMs (Felizardo et al, 2017). 
Graphical abstracts should enable reviewers to manipulate images to better understand 
the knowledge there represented (Oliveira and Levkowitz, 2003), since analyzing data 
in graphical format requires less cognitive effort from the reviewer to extract informa¬ 
tion. The main objective of the visualization is to help users explore and understand data 
(abstracts) (Keim, 2002). Visual representations allow users making use of their optical 
capabilities to strengthen the process of knowledge acquisition (Oliveira and Levkowitz, 
2003). In this context, CMs can be useful tools to summarize a complex structure of tex¬ 
tual information, contributing in identifying the most relevant information in an article. 
The results of this mapping will be used by us to support the construction of graphic 
abstracts based on CMs. 


5.1. Limitations 

The current review has some limitations. The study selection and data extraction steps 
were initially performed by just one of the authors, and thus some subjectivity could 
have been embedded. In order to reduce this subjectivity, the other authors performed 
these same steps over a sample of studies. The results of each reviewer were then com¬ 
pared in order to detect possible bias. 

As with all reviews it was limited by the search terms used and the electronic da¬ 
tabases included. However the studies discussed in this mapping provide a snapshot of 
empirical research on outcomes and impacts of existing research on CMs initiatives in 
Computer Science. We tried to overcome the limitations by using previous simulations 
in the selected databases. We just considered studies indexed by the selected electronic 
databases, and those obtained from by snowballing and direct search for works from 
researchers/research groups. The exclusion of other sources makes the review more re¬ 
peatable, but possibly some valuable studies may have been left out of our analysis. 


6. Conclusions 

A CM study gives an idea, in the early phases, of shortcomings in existing evidence, 
which becomes a basis for future studies Kitchenham and Charters (2007). This study 
presents a review of published literature on CMs initiatives in Computer Science in or¬ 
der to identify the state of the art. Seven research questions were defined and addressed 
investigating the following facets: (i) distribution of the selected studies over the years; 
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(ii) purposes to employ CM in Computer Science; (iii) different subareas that CMs have 
been applied; (iv) resources used to support the development of CMs; (v) guidelines or 
mechanism used to evaluate the CMs; (vi) main benefits reported on the use of CMs; and 
(vii) main problems reported on the use of CMs. 

The most notable point about the current review was the diversity of research on 
positive impacts and outcomes associated with the use of CMs in Computer Science. Al¬ 
though it is a tool considered old in other areas, CMs in Computer Science have shown 
to be very promising research area, since CMs helps to provide visual representation of 
knowledge structures to assistance in teaching and learning. 

The main contribution of this work is on making evident some aspects associated 
to the employment of CMs in Computer Science, in order to drive future research in 
this area. In this context, we highlight the following conclusions: (i) CMs in Computer 
Science is not a recent research, but has had a considerable increase in recent published 
studies; (ii) Assistance in Teaching and Learning and Knowledge Organization are the 
main purpose of applying CMs in Computer Science; (iii) there is a large diversity of the 
subareas of Computer Science in which CMs have been applied, such as Programming 
Language and Software Engineering; (iv) there seems to be no culture to evaluate CMs, 
which may impair the accuracy of CMs developed; (v) although CMs presents sev¬ 
eral benefits related to knowledge representation, sharing and pedagogical contribution, 
some problems also can be highlighted, such as cognitive overload, increased workload 
and scalability. 
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Wang et al. (2008) 

82 

Zheng et al. (2015) 

83 

Ladkin (2005) 

84 

Atapattu et al. (2015b) 

85 

Calvo et al. (2011) 

86 

Yamashita et al. (2009) 

87 

Roy (2008) 

88 

Krudysz and McClellan (2011) 

89 

Anohina and Grundspenkis (2007) 

90 

Hwang et al. (2011) 

91 

Lee et al. (2009) 

92 

Chen et al. (2001) 

93 

Anohina et al. (2007) 

94 

McClellan et al. (2004) 

95 

Valerio and Leake (2006) 

96 

Kommers and Stoyanova (2002) 

97 

Starr and de Oliveira (2010) 

98 

Chang et al. (2001) 

99 

Marshall et al. (2006a) 

100 

Chu et al. (2011) 

101 

Valerio et al. (2012) 

102 

Kaivola (2010) 

103 

Cimolino et al. (2003) 

104 

Liu et al. (2005) 

105 

Leake et al. (2004) 

106 

Reichherzer and Leake (2006) 

107 

Tseng et al. (2010) 

108 

Hwang et al. (2014) 


Appendix B. Mapping of individual studies to categories. 


The following Tables (Bl, B2, B3, B4, B5, B6) present the articles classifications (Sec¬ 
tion 3.4). We used the ID corresponding to each article created in the Appendix A. 


Table Bl 

Studies per purposes. 


Purposes 

Studies 

Learning Level Evaluation 

[1]; [2]; [13]; [15]; [17]; [18]; [20]; [22]; [35]; [40]; [42]; [45]; [60]; [64]; 
[67]; [69]; [72]; [92]; [93] 

Assistance in Teaching and 
Learning 

[3]; [4]; [6]; [11]; [14]; [18]; [19]; [21]; [24]; [27]; [28]; [29]; [30]; [31]; 
[32]; [33]; [34]; [35]; [36]; [37]; [38]; [39]; [41]; [44]; [46]; [48]; [49]; [51]; 
[52]; [54]; [58]; [59]; [61]; [63]; [65]; [66]; [69]; [70]; [71]; [72]; [78]; [79]; 
[87]; [99]; [85]; [88]; [77]; [93]; [90]; [91]; [94]; [96]; [97]; [98]; [100]; 
[101]; [102]; [103]; [104]; [108] 

Knowledge Organization 

[4]; [16]; [19]; [25]; [26]; [30]; [31]; [32]; [33]; [35]; [37]; [43]; [47]; [50]; 
[52]; [53]; [57]; [62]; [63]; [66]; [68]; [73]; [74]; [80]; [81]; [82]; [83]; [84]; 
[88]; [95]; [99]; [100]; [101]; [105]; [106]; [107] 

Software Visual Representation 

[5]; [7]; [8]; [9]; [10]; [23]; [86] 

General 

[55]; [56] 








Analyzing the Use of Concept Maps in Computer Science :... 
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Table B2 

Studies per Subarea. 

Subarea 

Studies 

Artificial Intelligence 

[2]; [49]; [95] 

Software Engineering 

[5]; [6]; [10]; [17]; [22]; [23]; [24]; [25]; [26]; [83]; [86] 

Database 

[44]; [58]; [94] 


Ontologies [1]; [26]; [56] 

Human-computer Interaction [50]; [67]; [80] 

Programming Languages [7]; [13]; [14]; [15]; [16]; [17]; [18]; [19]; [20]; [21] 



Table B3 

Studies per Resources. 

Resources 

Studies 

XML Schema 

[1]; [2]; [5]; [17]; [23]; [28]; [45]; [52]; [66]; [76]; [94] 

Concept Map Editor 

[1]; [2]; [3]; [5]; [6]; [7]; [9]; [10]; [15]; [17]; [21]; [23]; [24]; [25]; [42]; [44]; 
[45]; [47]; [51]; [52]; [58]; [59]; [60]; [65]; [71]; [73]; [76]; [84]; [85]; [87]; 
[93]; [96]; [102] 

Own system 

[3]; [4]; [8]; [11]; [12]; [14]; [16]; [19]; [20]; [27]; [28]; [29]; [30]; [31]; [32]; 
[34]; [33]; [35]; [36]; [37]; [38]; [39]; [40]; [48]; [49]; [52]; [53]; [54]; [57]; [61]; 
[63]; [68]; [72]; [74]; [76]; [78]; [79]; [80]; [81]; [82]; [88]; [90]; [91]; [93]; [92]; 
[94]; [97]; [98]; [99]; [100]; [101]; [103]; [104]; [105]; [106]; [107]; [108] 

Ontologies 

[23]; [25]; [26]; [41]; [50]; [55] [56]; [75]; [82]; [100] 

Collaborative Construction 

[27]; [28]; [29]; [31]; [39]; [40]; [56]; [99] 

Paper and Pen 

[7]; [13]; [15]; [30]; [31]; [34]; [51]; [58]; [99] 

Text Mining 

[16]; [43]; [66]; [95]; [100]; [101]; [107] 

Others 

[21]; [24]; [43]; [53]; [61]; [63]; [66]; [67]; [70]; [81]; [86]; [97] 

Not mention 

[46]; [69]; [77]; [83] 

Table B4 

Studies per Evaluation. 

Evaluation 

Studies 


Software to support in the comparison [1]; [5]; [45]; [49]; [56] 

Structural analysis [2]; [4]; [17]; [18]; [22]; [31]; [44]; [67]; [73]; [97] 

Relacional com mapa de expert [6]; [8]; [13]; [14]; [15]; [20]; [23]; [40]; [41]; [42]; [43]; [46]; [58]; 

[64]; [66]; [69]; [78]; [82]; [86]; [93]; [98]; [106] 
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Table B5 

Studies per Reported benefits . 

Reported benefits 

Studies 

Flexible structure and easy 
to understand 

[10]; [12]; [19]; [20]; [21]; [26]; [27]; [32]; [35]; [43]; [50]; [51]; [52]; [56]; [58]; 
[59]; [61]; [62]; [63]; [70]; [81]; [86]; [87]; [93]; [95]; [99]; [107] 

Pedagogical Contribuition 

[2]; [6]; [11]; [13]; [17]; [15]; [18]; [20]; [21]; [22]; [25]; [28]; [29]; [30]; [32]; 
[35]; [38]; [40]; [41]; [42]; [43]; [44]; [45]; [46]; [47]; [53]; [58]; [60]; [61]; [64]; 
[65]; [66]; [69]; [71]; [73]; [77]; [78]; [80]; [81]; [84]; [87]; [88]; [91]; [92]; [93]; 
[94]; [98]; [104]; [99]; [107]; [108] 

Knowledge Presentation 

[4]; [5]; [6]; [7]; [8]; [9]; [12] [16]; [19]; [20]; [23]; [24]; [26]; [29]; [30]; [31]; 
[33]; [35]; [36]; [39]; [42]; [43]; [48]; [52]; [57]; [61]; [66]; [67]; [68]; [73]; [74]; 
[77]; [81]; [83]; [84]; [87]; [95]; [97]; [99]; [101]; [103]; [105]; [106] 

Knowledge Sharing 

[5]; [6]; [10]; [11]; [12]; [16]; [24]; [26]; [27]; [29]; [30]; [32]; [33]; [35]; [36]; 
[37]; [38]; [40]; [43]; [45]; [46]; [48]; [49]; [52]; [54]; [61]; [65]; [66]; [72] [77]; 
[79]; [80]; [82]; [84]; [90]; [93]; [94]; [98]; [99]; [96]; [100]; [102]; [106] 

Table B6 

Studies per Reported Problems 

Reported Problems 

Studies 

Difficulty in creating CMs 

[13]; [22]; [43]; [66]; [93]; [98] 

Cognitive Overload 

[4]; [5]; [11]; [48]; [72]; [84]; [86]; [100]; [108] 

Increased Workload 

[5]; [15]; [43]; [48] [53]; [82]; [99] 

Scalability 

[4]; [5]; [11]; [33]; [68] [84] 

Others 

[4]; [6]; [13]; [21]; [26]; [34]; [41]; [54]; [67]; [69]; [76]; [78]; [86]; [93]; [107] 








